Informatik
Refine
Document Type
- Conference Proceeding (58)
- Doctoral Thesis (5)
- Master's Thesis (1)
Has Fulltext
- yes (64)
Is part of the Bibliography
- no (64)
Keywords
- clustering (3)
- data mining (3)
- k-means (3)
- law-enforcement (3)
- semi-supervised learning (3)
- Artificial Intelligence (1)
- Complexity Metrics (1)
- Electronic Commerce (1)
- Information Retrieval (1)
- Komplexitätsmetriken (1)
Institute
- Informatik (64)
- Fachbereich IV (1)
Automated machine learning represents the next generation of machine learning that involves efficiently identifying model hyperparameters and configurations that ensure decent generalization behavior beyond the training data. With a proper setup in place, considerable resources can be saved by practitioners and academics. Beyond naive approaches, e.g. random sampling or grid search, sequential model-based optimization has been at the forefront of solutions that attempt to optimize the black-box function representing the generalization surface, for example, the validation loss.
With the abundance of data and algorithm evaluations being available, transfer learning techniques and meta-knowledge can be utilized to further expedite hyperparameter optimization. In this thesis, we cover 4 ways in which meta-knowledge can be leveraged to improve hyperparameter optimization.
In the first part, we present two large-scale meta-datasets, i.e. a collection of hyperparameters and their respective response for a machine learning algorithm trained on several datasets. We describe in detail the implementation details and descriptive analytics that highlight the heterogeneity of the resulting response surface. The two meta-datasets are used as benchmark datasets upon which the subsequent methods developed in this thesis have been empirically evaluated.
In the second part, we introduce the first work that automates the process of learning meta-features, i.e. dataset characteristics, directly from the dataset distribution. Previously, meta-features required expert-domain knowledge and a lot of engineering to properly represent datasets as entities for a meta-learning task. Following this work, we integrate the meta-feature extractor as a module in the machine learning algorithm, and optimize it jointly for the meta-learning task, further promoting the benefits of differentiable meta-features. Finally, we carry over the concept of meta-feature learning in the absence of the underlying dataset. Specifically, we design a deep Gaussian kernel that allows for a richer representation of the attributes via non-linear transformation. The resulting surrogate is conditioned on landmark meta-features extracted from the history of task-specific evaluations.
In the third part, we formulate the problem of hyperparameter optimization as a Markov Decision Process. As such, we introduce the first paper on hyperparameter optimization in a reinforcement learning framework and define a novel transferable policy that acts as an acquisition function for hyperparameter optimization. Furthermore, we study the impact of planning in hyperparameter optimization through a novel non-myopic acquisition function.
Finally, we present hyperparameter optimization in a zero-shot setting. In contrast to sequential model-based optimization, the fastest way for HPO is by learning a zero-shot approach, that identifies the best configuration with a single trial. Our Zap-HPO approach outperforms the state-of-the-art in algorithm selection for deep learning pipelines that comprise a machine learning algorithm and its associated hyperparameters, given simple meta-features.
Recommender systems have been deployed in many diverse settings, and they aim to provide a personalized ranked list of items to users that they are likely to interact with. In order to provide an accurate list of items, models need to capture various aspects of the users' profiles, behaviors, and items' dynamics. Depending on the recommendation settings, these aspects can be mined from the different auxiliary information sources that might be readily available in these settings as side information. The more aspects being covered, the more accurate the learned user and item representations will be, improving prediction performance and overcoming various challenges such as sparse interaction data.
These auxiliary information sources might contain static attributes related to the users' and items' profiles or contain historical multi-relational implicit interactions between users and items, users and users, and items and items such as clicks, views, bought-together, and friendships. These interactions can be exploited to capture complex implicit relations that are usually not visible if the model only focuses on one user-item relationship.
Besides attributes and interaction data, auxiliary information might also contain contextual information that accompanies the interaction data, such as timestamps and locations. Incorporating such contextual information allows the models to comprehend the dynamics of users and items and learn the influence of time and environment.
In this thesis, we present four ways in which auxiliary information can be leveraged to improve the prediction performance of recommender systems and allow them to overcome many challenges.
Firstly we introduce an attribute-ware co-embedding model that can leverage user and item attributes along with a set of graph-based features for rating prediction. In particular, the model treats the user-item relation as a bipartite graph and constructs generic user and item attributes via the Laplacian of the co-occurrence graph. We also demonstrate that our attribute-ware model outperforms existing state-of-the-art attribute-aware recommender systems.
Next, we extend the model to handle different multi-relational interactions to overcome the challenges of having few and sparse interaction data between users and items. First, we extend the model by adding the capability to capture multi-relational interactions between the same entity types, particularly between users and users and between items and items. This is done by simultaneously scoring the different relations using a weighted joint loss. Later, we extend the model further by including the ability to accommodate different user and item interactions simultaneously by having an independent scoring function for each interaction type. The later extension allows the model to be employed in scenarios where the main relation between users and items is extremely sparse such as in auction settings which pose a significant challenge to traditional and state-of-the-art models.
Additionally, we introduce a sequential context and attribute-aware model that captures users' and items' dynamics through their sequential interaction patterns and their timestamps. The model can also capture various aspects of the users' and items' profiles through their static attributes and content information.
Finally, in the end, we also present a framework for optimizing ranking metrics directly, such as the Normalized Discounted Cumulative Gain (NDCG) using surrogate losses as an additional way of improving the models' performance.
Hintergrund: Die Software-Produktlinienentwicklung ermöglicht eine kostengünstige und effiziente Entwicklung von Produktfamilien bei gesteigerter Qualität im Vergleich zur Einzelsystementwicklung. Dieses wird durch die Einführung von Variabilitätsmechanismen ermöglicht, welche eine hohe Anpassbarkeit der Produkte an verschiedene Kundenbedürfnisse ermöglichen. Allerdings erhöhen diese Variabilitätsmechanismen auch die Komplexität, da Entwickler das Zusammenwirken der Komponenten für verschiedene Produktvarianten berücksichtigen müssen. Daher wurden zur Qualitätssicherung von Software-Produktlinien neue Analysemethoden und -strategien entwickelt, darunter auch variabilitätsbasierte Code-Metriken. Ziel dieser Metriken ist es, unnötige Komplexität zu vermeiden und frühzeitig besonders fehleranfälligen Code zu identifizieren, um diesen zusätzlichen Qualitätsmaßnahmen unterziehen zu können. Unsere systematische Literaturstudie zu diesem Thema zeigt jedoch, dass der Nutzen dieser variabilitätsbasierten Code-Metriken nur in wenigen Fällen evaluiert wurde.
Ziel: Diese Arbeit untersucht inwieweit variabilitätsbasierte Code-Metriken zur Qualitätssteigerung von Software-Produktlinien genutzt werden können. Dazu wird betrachtet, ob sich mit Hilfe empirischer Untersuchungen Entwicklungsrichtlinien zur proaktiven Vermeidung von Komplexität und damit verbundenen Fehlern ableiten lassen. Der Fokus liegt auf der Analyse, ob sich die betrachteten Metriken zur Identifikation von potentiell fehleranfälligeren Code nutzen lassen. Dies umfasst sowohl die univariate Verwendung einzelner Metriken als auch den Aufbau von Vorhersagemodellen mit Verfahren des maschinellen Lernens. Dabei wird auch untersucht, ob die speziell für die Software-Produktlinienentwicklung konzipierten variabilitätsbasierten Code-Metriken einen Mehrwert gegenüber etablierten Metriken der Einzelsystementwicklung bieten.
Methodik: Es findet eine empirische Untersuchung von 692 Einzelsystem- und variabilitätsbasierte Code-Metriken auf dem Linux-Kernel statt. Dazu wird zunächst analysiert, inwieweit die Messwerte der Metriken mit Kompilierfehlern und Sicherheitslücken korreliert sind, welche von den Entwicklern übersehen wurden und so erst nach dem Commit bzw. nach dem Release entdeckt wurden. Darüber hinaus werden die Metriken bezüglich der gemessenen Eigenschaften gruppiert und mit vier Verfahren des maschinellen Lernens eine Identifikation der fehleranfälligen Codestellen erprobt, um so den Nutzen verschiedener Codeeigenschaften beurteilen zu können.
Ergebnisse und Schlussfolgerung: Auch wenn für einen Großteil der Metriken ein signifikanter Zusammenhang zwischen den Messwerten und fehleranfälligen Codestellen nachgewiesen werden kann, so zeigt sich, dass univariate Verfahren für die Praxis untauglich sind. Auf Grund der starken Klassenimbalance von nur 1,5% defekten Codefunktionen (Kompilierfehler), bzw. 0,4% Codefunktionen mit nachgewiesenen Sicherheitslücken, lassen sich bei der Verwendung einer einzelnen Metrik nur F1-Werte unterhalb von 0,073 erzielen. Mangels alternativer Implementierungen lassen sich so, entgegen den Aussagen einiger Veröffentlichungen, auch keine Entwicklungsempfehlungen ableiten. Hingegen können variabilitätsbasierte Code-Metriken, insofern sie auch die Variabilität verbundener Artefakte mit berücksichtigen, erfolgreich zur Fehlervorhersage genutzt werden.
So lässt sich beispielsweise bei Verwendung von Random Forest F1-Werte von 0,667 (Kompilierfehler), bzw. 0,711 (Sicherheitslücken), erzielen.
Recommender systems are personalized information systems that learn individual preferences from interacting with users. Recommender systems use machine learning techniques to compute suggestions for the users. Supervised machine learning relies on optimizing for a suitable objective function. Suitability means here that the function actually reflects what users and operators consider to be a good system performance. Most of the academic literature on recommendation is about rating prediction. For two reasons, this is not the most practically relevant prediction task in the area of recommender systems: First, the important question is not how much a user will express to like a given item (by the rating), but rather which items a user will like. Second, obtaining explicit preference information like ratings requires additional actions from the side of the user, which always comes at a cost. Implicit feedback in the form of purchases, viewing times, clicks, etc., on the other hand, is abundant anyway. Very often, this implicit feedback is only present in the form of positive expressions of preference. In this work, we primarily consider item recommendation from positive-only feedback. A particular problem is the suggestion of new items -- items that have no interaction data associated with them yet. This is an example of a cold-start scenario in recommender systems. Collaborative models like matrix factorization rely on interaction data to make predictions. We augment a matrix factorization model for item recommendation with a mechanism to estimate the latent factors of new items from their attributes (e.g. descriptive keywords). In particular, we demonstrate that optimizing the latent factor estimation with regard to the overall loss of the item recommendation task is superior to optimizing it with regard to the prediction error on the latent factors. The idea of estimating latent factors from attributes can be extended to other tasks (new users, rating prediction) and prediction models, yielding a general framework to deal with cold-start scenarios. We also adapt the Bayesian Personalized Ranking (BPR) framework, which is state of the art in item recommendation, to a setting where more popular items are more frequently encountered when making predictions. By generalizing even more, we get Weighted Bayesian Personalized Ranking, an extension of BPR that allows importance weights to be placed on specific users and items. All method contributions are supported by experiments using large-scale real-life datasets from various application areas like movie recommendation and music recommendation. The software used for the experiments has been released as part of an efficient and scalable free software package.
Predicting student performance (PSP) is an important task in Student Modeling where we would like to know whether the students solve the given problems (tasks) correctly, so that we can understand how the students learn, provide them early feedbacks, and help them getting better in studying. This thesis introduces several approaches, which mainly base on state-ofthe- art techniques in Recommender Systems (RS), for student modeling, especially for PSP. First, we formulate the PSP problem and show how to map this problem to rating prediction task in RS and to forecasting problem. Second, we propose using latent factor models, e.g., matrix factorization, for student modeling. These models could implicitly take into account the student and task latent factors (e.g., slip and guess) as well as student effect/bias and task effect/bias. Moreover, there is a fact that similar students may have similar performances, we suggest using k-nearest neighbors collaborative filtering to take into account the correlations between the students and the tasks. Third, in student's problem solving, each student performs several tasks, and each task requires one or many skills, while the students are also required to master the skills that they have learned. We propose to exploit such multiple relationships by using multi-relational matrix factorization approach. Fourth, as the student performance (student knowledge) cumulates and improves over time, a trend line could be observed in his/her performance. Similar to time series, for solving this problem, forecasting techniques would be reasonable choices. Furthermore, it is well-know that student (human) knowledge is diverse, thus, thought and performance of one student may differ from another one. To cope with these aspects, we propose personalized forecasting methods which use the past performances of individual student to forecast his/her own future performance. Fifth, since student knowledge changes over time, temporal/sequential information would be an important factor in PSP. We propose tensor factorization methods to model both the student/task latent factors and the sequential/temporal effects. Sixth, we open an issue for recommendation in e-learning, that is, recommending the tasks to the students. This approach can tackle existing issues in the literature since we can recommend the tasks to the students using their performance instead of their preference. Based on student performance, we can recommend suitable tasks to the students by filtering out the tasks that are too easy or too hard, or both, depending on the system goal. Furthermore, we propose using context-aware factorization approach to utilize multiple interactions between the students and the tasks. Seventh, we discover a characteristic in student performance data, namely class imbalance problem, i.e., the number of correct solutions are higher than the number of incorrect solutions, which may hinder classifiers' performance. To tackle this problem, we introduce several methods as well as introducing a new evaluation measure for learning from imbalanced data. Finally, we validate the proposed methods by many experiments. We compare them with other state-of-the-art methods and empirically show that, in most of the cases, the proposed methods can improve the prediction results. We therefore conclude that our approaches would be reasonable choices for student modeling, especially for predicting student performance. Last but not least, we raise some open issues for the future research in this area.
In this paper, the idea of ubiquitous information retrieval is presented in a storytelling manner. Starting from a rough review of information retrieval system usage, some empirical hints on IR in everyday life are given. Ch. 4 explores the heterogeneity of interaction with IRS for one day in the life of a (common search engine) user. In ch. 5 summarizes these observations and suggests research approaches for modelling information retrieval as an essential component of interaction in the information society.
Im vorliegenden Beitrag werden benutzerpartizipative Verfahren im Rahmen des Datenbankentwurfs für ein Informationssystem vorgestellt. Dabei wird aufgezeigt, wie Extreme Programming als zentraler Ansatz der agilen Software Entwicklung die synergetische Verflechtung des traditionell technologiebetriebenen Software Engineering (SE) mit benutzerzentrierten Verfahren des User-Centered Design (UCD) ermöglichen kann und welche Mehrwerte sich daraus ergeben. Da insbesondere die Kommunikation zwischen Systementwicklern und Experten im vorgestellten Projekt einen hohen Stellenwert einnahm, werden entsprechende Vorgehensweisen, aufgetretene Probleme sowie Lösungsansätze in der Anforderungsanalyse diskutiert. Der Einsatz von Interview- und Beobachtungstechniken wird dabei am Beispiel der Erfassung des Car Multimedia Anwendungsfeldes zum Zweck der Daten- und Systemmodellierung verdeutlicht.
The learners’ motivation has an impact on the quality of learning, especially in e-Learning environments. Most of these environments store data about the learner’s actions in log files. Logging the users’ interactions in educational systems gives the possibility to track their actions at a refined level of detail. Data mining and machine learning techniques can “give meaning” to these data and provide valuable information for learning improvement. An area where improvement is absolutely necessary and of great importance is motivation, known to be an essential factor for preventing attrition in e-Learning. In this paper we investigate if the log files data analysis can be used to estimate the motivational level of the learner. A decision tree is build from a limited number of log files from a web-based learning environment. The results suggest that time spent reading is an important factor for predicting motivation; also, performance in tests was found to be a relevant indicator of the motivational level.
Personalization involves the process of gathering user-specific information during interaction with the user, which is then used to deliver appropriate results to the user’s needs. This paper presents a statistical method that learns the user interests by collecting evidence from his search history. The method focuses on the use of both user relevance point of view on familiar words in order to infer and express his interests and the use of a correlation metric measure in order to update them.
Class binarizations are effective methods that break multi-class problem down into several 2-class or binary problems to improve weak learners. This paper analyzes which effects these methods have if we choose a Naive Bayes learner for the base classifier. We consider the known unordered and pairwise class binarizations and propose an alternative approach for a pairwise calculation of a modified Naive Bayes classifier.
